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ABSTRACT 

This paper is a case study about the role of 
evaluation and its effect on a foster care program's adoption 
services. It presents examples of the dilemmas often faced by the 
evaluator who must identify problems and design action plans to 
correct program deficiencies. HeartShare Human Services of New York 
is an 80-year-old nonprofit agency that provides foster care among 
its other ser ices. The program is committed to developing permanency 
plans, including arranging adoption when possible. The Child Welfare 
Administration (CWA) of New York (New York) has evaluated the 
program's performance over time. In 1991 CWA developed a new system 
for assessing performance that weighted milestones in the adoption 
process and established a target of completing at least 507o of 
adoptions within 27 months. The inability of the program to meet the 
new goals and changes to the adoption assessment system resulted in 
several organizational changes to improve performance and ensure that 
information flowed more smoothly within the agency. Program 
evaluation, as the case study illustrates, resulted in improved 
performance in this social services program. (SLD) 
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USING PROGRAM EVALUATION 
TO AFFECT POSITIVE CHANGE IN PROGRAM PERFORMANCE 



INTRODUCTION 



Over the pasty twenty years , program evaluation has come under 
increasing scrutiny for its general usefulness in affecting 
positive change in program performance. Several theorists, such as 
Patton and White, have begun to question the overall usefulness of 
program evaluation because it fails to meet management expectations 
or affect program performance. 



This paper is a case study about the role of evaluation and 
its affect on a Foster Care Program's Adoption Services. It 
presents an example of dilemmas often faced by the evaluator whose 
task is to identify problems and design action plans that will 
correct program deficiencies. Once the evaluator accomplishes both 
tasks, management assumes it will improve program performance. 



* In presenting this study, the author will provide a historical 
context that will explain systematic issues affecting the role of 
an internal evaluator. The second part will focus on various 
methods employed to gather information and systems developed to 
affect program performance. The final section will summarize issues 
that affected decisions made during the course of the project and 
how it strengthened the role of evaluation in an organizational 
setting. 
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HISTORICAL BACKGROUND 



HeartShare Human Services of New York is an eighty year old 
voluntary non-profit agency that provides services in the areas of 
aids case management, developmental disabilities, foster care and 
preventive services. The agency has two primary funding sources. 
New York City's Child Welfare Administration funds our foster care 
and preventive services while New York State's Department of Social 
Services and Office of Mental Retardation and Disabilities funds 
our aids case management and developmental disabilities programs. 



The Foster Care Program, which is a major component cf the 
HeartShare Service Delivery System, has two major purposes. First, 
its primary focus is to provide temporary placement for children 
whose parents are experiencing a crisis or because of an abuse or 
neglect allegation with the ultimate goal of returning the child 
home. Second, if it 'is determined the child will not return home, 
the program will either change the discharge plan to adoption or 
independent living. The fundamental goal of the program is to do 
what is in the best interest of the child by providing permanency 
planning. More specifically, the program is committed to 
identifying the problem that brought the child into care; 
developing a permanency plan that will determine the appropriate 
placement? and discharging the child out of the foster care system 
within 18 or 24 months dependent on child's age. 

Since CWA has primary responsibility for monitoring program 
performance and setting standards for accountability purposes , 
HeartShare has maintained an internal evaluation system. This 
internal evaluation component resides in the Management Information 
Services Department. One of the unit's functions include program 
evaluation and quality assurance activities in the Foster Care 
Program. The Director reports to the Associate Executive Director 
for Support Services who in turn reports directly to the Executive 
Director. Although the MIS Director reports to the Associate 
Executive Director for Support Services, he has access to the 
Executive Director as well as the Associate Executive Director for 
Children and Families Services who oversees the Foster Care 
Program. Overall, the Director is authorized to conduct ongoing 
evaluation and quality assurance activities in the Foster 
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Care Program to ensure program compliance with regulatory 
standards and providing quality service to its 1 client population. 



As previously stated, the Child Welfare Administration is 
responsible for funding the foster care program; setting regulatory 
standards? and monitoring program performance. Prior to 1991, CWA 
had a program assessment unit for collecting data; making on-site 
visits; analyzing results and writing reports to ensure program 
compliance. The primary way CWA monitored adoption performance was 
twofold. First, they would utilize information from the state 
computerized case management system. This system contained 
information on all movements regarding children in care* Second, 
they would generate reports showing agency success and failure 
rates in meeting target dates. Ultimately, these reports assessed 
agency performance in meeting system standards. 



In determining agency performance, CWA utilized several 
milestones. They were: 



(1) setting adoption as a goal within 18 months of initial 
placement date; (2) filing a petition with the court to terminate 
parental rights within four months of goal approval date; (3) 
freeing the child within eight months of petition filing date; (4) 
placing a child in an adoptive home within four months of freed 
date; (5) finalizing adoption within six months of placement date 
if child was being adopted by his or her foster parent or twelve 
months if a child was placed in a pre adoptive home; and (6) 
completing the entire adoption process within 2 7 months of original 
goal set date. 



D 



When assessing scores, milestones were divided into several 
variables. There was one for setting goal. The second grouped 
petition filing, freeing, placing and finalization milestones into 
another variable known as adoption milestones. The third variable 
was completing c.doptions within 27 months. If the agency failed to 
take no action i:o meet target dates for petition filing, freeing, 
placing and finalizing , they were grouped into another variable 
known as adoption milestone exceptions. 

In determining scores , CWA used two review periods. They were 
January to June and July to December. Pass or failure was 
determined by target dates. For most variables, pass or failure was 
determined by initial target date and adoption milestone exceptions 
were assessed by agencies ability to complete tasks within a six 
month review period after missing original target dates. 

Since program assessment was driven by a computerized case 
management system , it was very important to monitor information 
flow to ensure accuracy of data entry on each child 1 s movement in 
the state computer and having an internal case management system 
that would advise workers of actions required to meet target dates 
for each adoption milestone. 

From the period 1989 to 1991 , MIS would generate reports on 
all milestones showing what had to be done in order to be in 
compliance. Further , MIS prepared quarterly reports showing how 
well the program was meeting target dates by time period and site. 
As a result of these reports, Directors knew how well the program 
was doing for each milestone and which sites were more efficient at 
meeting standards. During this time, Directors felt it was valuable 
to have these reports. However, it was decided that Directors would 
act on deficiencies as opposed to having the evaluator play a pro 
active role in correcting problem areas. 

Over the course of time, reports showed there were major 
problems in meeting filing dates and freeing children. Further, 
there were ongoing inconsistencies between information that was in 
the internal and state case management system. 
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In order to correct the later issue, the MIS Director met with 
the Adoption Director to develop a system for reconciling 
discrepancies. As a result of the discussion, a system was put in 
place but no action was taken to utilize the system. In general, 
neither the Adoption Director nor her staff wanted to deal with the 
issue. As a result, there was no effort made to correct the 
problem. 



In 1991, CWA developed a new system for assessing performance. 
Milestones were given weighted scores and agencies were asked to 
complete at least 50% of all adoptions that were coming up to the 
27 month period or were past the 27 month target date. As a result, 
the agency was under increasing pressure to improve total numbers 
since the program finalized only 26 adoptions for the 90/91 fiscal 
year. Further, CWA developed a scoring mechanism that was now based 
on a weighted score, system standard and changes in milestone 
target dates for freeing children. The milestones, that were 
considered in assessing adoption performance, were as follows: 



(1) setting adoption as a goal within 18 months; (2) filing a 
petition within four months of goal set date; (3) freeing child 
within one year of goal set date; (4) placing child within four 
months of free date; (5) achieving adoption within 27 months of 
goal set date; and (6) total number of adoptions reaching 27 month 
or past 27 month. 



As for the scoring system, you received up to 4 points for 
finalizing at least fifty percent of all adoptions due for the 
fiscal period ; 1.5 for petition filing and placement dates since 
the agency had total control over these milestones; while the 
remaining milestones had a weighed score of 1.0 each. These scores 
were totaled and a figure was arrived at that determined whether 
you were satisfactory, needs improvement or unsatisfactory. 
In addition, numbers were cumulative in calculating system 
standards and agency scores. Therefore, if you were an agency that 
failed to complete a significant number of milestone dates, it was 
reflected in your score. Therefore, it became increasingly 
important to clear up past due milestone dates. 
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EVALUATION SYSTEM 



As a result of changes to CWA f s adoption assessment system and 
program's inability to improve adoption performance, the Executive 
Director instituted several organizational changes. First, the MIS 
Director was given the adoption legal unit in order to clear up 
discrepancies in milestone information that existed between the 
state and internal system* Second, the MIS Director was asked to 
work closely with the program's administrative staff in order to 
monitor performance and ensure information flowed more smoothly 
between agency workers and attorneys responsible ^for terminating 
rights and finalizing adoptions. 



In order to improve performance, several systems were 
implemented immediately. 



First, MIS personal was given responsibility for imputing all 
information into both the state and internal information systems. 
Second, the compliance coordinator was given the task to conduct 
ongoing audits to ensure accuracy of data input in both systems. 
This system included a periodic review of adoption cases to ensure 
there were no discrepancies between both systems and if there were 
to work closely with the Assistant QA Director, Program and CWA 
staff to correct problem immediately. 



Second, the Assistant Director for Quality Assurance worked 
closely with the Adoption Coordinator to identify adoption cases 
that would be completed by the end of the fiscal year. Once these 
cases were identified, a computer data base showing information 
currently in file and outstanding documents that were needed for 
completing adoptions was developed, with reports being sent to 
administrative staff on a bi-weekly and currently monthly schedule. 
In these reports, each case listed information needed to complete 
finalization and identified individuals responsible for completing 
tasks. If tasks were not completed, reasons were requested and 
systems were identified to expedite paperwork. 
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Third, the Assistant Director for Quality Assurance has weekly 
meetings with agency attorney to review cases and identify tasks 
necessary to expedite cases. In addition, monthly meetings were 
held with program and legal staff to review progress of cases and 
determine tasks necessary for expediting f inalizations . 

Fourth, quarterly reports were prepared for administrative 
staff and monthly meetings were held to review results and develop 
action plans to correct deficiencies. 



Fifth, MIS staff developed close working relationships with 
attorneys hired by families to finalize adoptions and court clerks 
responsible for calendaring cases in order to enlist their support 
in expediting cases through the system. 



As a result of these changes, the agency improved their 
petition filing performance by 3 0% and increased their adoption 
f inalizations from 26 for fiscal year 90/91 to 106 for fiscal year 
92/93. 



SUMMARY 



This paper provides an overview of how evaluation was used to 
affect program perfprmance. As a result of this experience, several 
issues arose that affected decisions made during the course of the 
project and enhanced the learning process on the appropriate 
utilization of evaluation to affect program performance. 

First, evaluation has to make sense and meet the needs of the 
end user. When the MIS Director was originally hired by the agency, 
there was a QA system already in place. For the most part, it 
consisted of the QA Director reading case records; preparing 
rebuttal reports to CWA Audits; and copying the previous years 
action plan to show how the agency would correct program 
deficiencies. In reviewing the system, the MIS Director came to 
several conclusions . 
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(A) The pre reports, that were designed to show directors and 
supervisors what, was coming due for each milestone, were not used 
properly nor user friendly. As a result, there was a need to 
develop a case management system that would provide the necessary 
information by supervisor and worker. 

(B) Post reports were for the most part erratic and failed to 
identify location of system breakdowns. The new system was designed 
to provide information that highlights program performance by site. 
By using this process, the agency was able to pinpoint location of 
system breakdowns and target corrective action plans to that 
specific area. 

(C) In the past, QA prepared reports and program people had 
primary responsibility to decide how best to utilize results. Now, 
the system requires ongoing sessions with administrative staff who 

• respond to MIS reports by showing what actions will be taken to 
resolve deficiencies or explain why there cannot be improvement in 
performance. 

Second, Evaluation is meaningless unless it has the full 
support of the Executive Staff. Unlike the past, the Executive 
Director as well as the Associate Executive Director for Children 
and Families Services gave the MIS Director a great deal of freedom 
in developing systems for correcting deficiencies. Therefore, the 
MIS Director has periodic meetings with the Executive Director 
and/or Associate Executive Director to discuss problems and systems 
that would be designed to rectify problems. 

Third, evaluation relies on meaningful information. In the 
past, the Adoption Director was responsible for having adoption 
information imputed into the state system and passing it along to 
MIS . Since the Adoption Director was not a systems person , the 
information flow became inadequate which had an impact on the 
quality of reports generated by MIS. With the new system, MIS has 
total control over adoption information flow ensuring the quality 
of data used for deveJ oping pre and post reports . Further, it 
becomes easier to isolate breakdowns that may occur within the 
system. 
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Fourth, evaluation can only, succeed if all individuals have a 
clear understanding of roles and a willingness to work together to 
correct deficiencies. In the past, MIS worked in isolation from 
program and legal staff. The new system encourages ongoing 
dialogues between MIS and Program staff as well as MIS and Agency 
Attorneys so everyone has a clear understanding of their roles and 
actions necessary to complete tasks that will expedite the adoption 
process. 

In conclusion, this case study is an example of one approach 
for utilizing evaluation to affect program performance. It clearly 
shows how to use an evaluation system that adapts to the situation; 
is flexible; user friendly; and meets the needs of the end user. 
Overall, the system not only had a positive impact on increasing 
the total number of adoptions but it also helped in strengthening 
the permanency planning system which led to a total of 2 62 children 
either being returned to their parent; adopted; or discharged to 
independent living in the 92/93 fiscal year. Further, it fostered 
a positive relationship between MIS and Program staff by helping 
both groups focus on the primary goal of developing an appropriate 
permanency plan for each child who came into the program's care. 
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